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ABSTRACT 

This handbook spells out the steps necessary in an 
effective evaluation of secondary/transition interventin projects for 
handicapped youth. Initial steps in the evaluation process include 
describing the evaluation setting, identifying evaluation audiences, 
stating the evaluation rationale, determining evaluation questions, 
and identifying evaluation constraints. Subsequently, planning and 
implementing the evaluation involves determining evaluation 
approaches and methods, collecting the data, and analyzing the data. 
Well-written reports are then needed to document program results, 
provide a framework for replication and/or extension of the program, 
respond to stakeholder needs, and serve as a point of departure for 
program improvement. Throughout this evaluation guide, examples are 
included from current projects to illustrate how evaluation issues 
can be approached and resolved. Appendices contain worksheets and an 
annotated bibliography of 23 items. (JDD) 
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FOREWORD 



This Evaluation Handbook for secondary/transition intervention projects 
is designed to serve as a guide for project staff who conduct evaluations 
as part of their responsibilities in meeting grant regulations for account- 
ability. In addition, technical assistance (TA) staff hope that the Hand- 
book will serve as a resource guide for projects as they assess their 
effectiveness, and improve their programs or components, and report on 
their programs to their many constituencies. 

The authors wish to acknowledge the assistance provided them by the 

following individuals who graciously agreed to review this manuscript and 

provided many recommendations and insights which contributed significantly 

to the Handbook. Those to whom we are indebted include! 

Dr. Frank R. Rusch, Dr. L. Allen Phelps and Dr. Lizanne DeStefano, 
Transition Institute, university of Illinois; 

Ms. Kay Holjes, Employment Opportunities, Inc., Durham, N.C.; 

Dr. Earl E. Davis, Univ. of Tennessee at Chattanooga; 

Ms. Susan Gurganus, North Carolina Department of Public 
Instruction 

Vfe also wish to acknowledge the assistance of Jeri Conklin and 
Karla Colegrove in the preparation of the manuscript and Marie D. Eldridge, 
Director of the Center for Educational Studies, for her extensive review 
and edit. 
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CHAPTER 1 
INTRCDUCTION 

On Decentoer 2, 1983, the United Sta:es Congress revised and extended the 
Education of the Hancicapped Act through a series of amendments, Public Law 
(PL) 98-199. These amendments authorized grants for secondary education and 
transitional services for handicapped youth (Sec. 626), which were to include 
demonstration models, exemplary service delivery models, and cooperative 
models between educational agencies and adult service agencies. These grants 
were to be coordinated with projects operating undeo. Section 311 of the 
Rehabilitation Art of 1973. Postsecondary education proarams also were 
eligible to receive grants under Section 625 of PL 98-199, as were research 
and datonstration projects to help special education personnel, related 
services personnel, and "other appropriate persons, including parents, improve 
the education and related services for handicapped children and youth" 
(PL 98-199, Sec. 641. a). These projects represent secondary/transition 
intervention projects (model programs) for which this Evaluation Handbook has 
been designed. 

The legislation for the model programs required program evaluations (PL 
98-199, Sec. 618); and in subsequent regulations for grant awards, it was 
stated that the Secretary would review each application to assess the quality 
of its evaluation plan ( Federal Register, Vol. 49, Kb. 123, Monday, June 25, 
1984, p. 25993) . These regulations reinforced the importance of evaluation, 
not only for reporting to Congress but also for project management and 
planning. This Handbook is designed to facilitate project responsiveness to 
the federal government and project capability regarding project evaluation. 

Evaluation is the process of determining the worth of something. It 
includes obtaining information for use in judging the overall worth of a 
project or of specific project components, objectives, etc. Evaluation also 
enables program personnel to assess the extent to which program goals and 
objectives have been attained. Additionally, it provides information for 
iitproviiig less successful elements of a project and for extending effective 
practices. Evaluation can assist in crmnunicating inpact information to 
people interested in the program's outcomes, whether they be youth enployirent, 
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postsecondary training, dissemination of strategies, or cooperative agreement 
among agencies. And, a well-documented ^valuation can provide a basis for 
replication of the program. 

Given the multitude of potential uses of an evaluation, it should be clear 
that there is no such thing as the "perfect" evaluation. Rather, like pro- 
viding individualized intervention to' a specific project client /consumer, 
evaluations need to be individually tailored to each project's evaluation 
needs, goals, abilities, and resources. This means that project staff must 
determine what they want/need from an evaluation and balance this with the 
personnel, time, and budget available for an evaluation. The result may be a 
one- or two-day effort, or a three-year process in which all staff associated 
with the project are involved. Whatever the scope and/or complexity, evalua- 
tion is both important and difficult to do well. Thus, we (the authors) hope 
that this Handbook is of assistance to project staff as they face the chal- 
lenge of conducting a useful evaluation within the practical constraints of 
their mission and funding. 

Since this Evaluation Handbook is designed specifically for transition 
projects, we have included examples from current projects to assist the eval- 
uator in the translation for the processes we discuss to their individual 
setting. The examples are presented in the following format to afford ready 
access as well as to avoid discontinuity in the text itself. 



EVALUATION SETTING: PROJECT RATIONALE 

In Minneapolis a postsecondary project is 
based on the belief that learning disabled 
adults in mainstream postsecondary settings 
need long-term modifications to their writing 
process. . . 



PROJECT STAKEHOLDERS 

The audience/stakeholders identified by 
staff at the Fort Collins project are: OSERS 
staff, state legislators, state agency staff 



EVALUATION RATIONALE 

In Blocmington, Indiana, Project COMPETE 
is a cooperative project that is designed to 
• • • 

The purpose of the evaluation is to 
answer two related questions: (1) . . . 
(2) . . . 

\ 

These examples will illustrate how project evaluation issues can be 
approached and resolved. Evaluators reading this material will need to tailor 
it to their particular project setting and evaluation needs, draw upon their 
resources, and not be reluctant to question and be innovative. This becomes 
particularly inportant when programs are modified* such itodifications 
generally necessitate modifications in the evaluation plan. It is inportant 
to note that evaluations, more often than not, emerge by way of an iterative 
process. Again, there is no such thing as a "perfect" evaluation. 

Nonetheless, there are systematic logical steps that are generally recog- 
nized as essential in the evaluation process. These are presented schema- 
tically in Figure 1 on the following page. Topics included in the process are 
discussed with major enphasis on 

o Focusing the evaluation, 

o Planning and inplementing the evaluation, and 
o Reporting and using the evaluation results. 
To seme, with limited resources for evaluation, these ten distinct steps 
may appear formidable. Hcwever, knowledge of the systematic steps involved in 
a formal evaluation will undoubtedly assist those projects which do not have 
specific or large evaluation budgets to sit down and document what services 
ware provided., who was served, and what outcomes- were attained. 

Chapter 2 addresses the initial steps in the evaluation process. In order- 
to focus an ev<duation so that the results of the evaluation are the irost 
relevant and useful, certain sequential steps should be taken. These steps 
include a thorough description of the evaluation settings, the identification 
of the audience/stakeholders for the evaluation results, a statement of the 

3 
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Figure 1. Steps in the Evaluation Process 
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Analysing Data 



Reporting Evaluation 
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Using Evaluation 
Findings 



Reporting and 
Using the 
Evaluation Results 
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evaluation rationale, a determination of the nost relevant evaluation ques- 
tions, and the identification of constraints impinging upon the evaluation 
process. 

Chapter 3 discusses considerations and guidelines for planning and imple- 
menting the evaluation. An overview of how to select and apply the nost 
appropriate method of evaluation is presented. The development of an evalua- 
tion management plan to structure the evaluation process and suggestions 
rega r di n g data collection are found in this chapter. 

Chapter 4 discusses planning the presentation of results, reporting 
results, and using the results of the evaluation. The best executed evalua- 
tion is inccnplete without well written reports to document program results, 
to provide a framework for replication and/or extension of the program, to 
respond to stakeholder needs, and to serve as point of departure for program 
improvement. 

An annotated bibliography is provided in Appendix A; worksheets for the 
material covered in chapters 2 through 4 are presented in Appendices B (Chap- 
ter 2), C (Chapter 3), and D (Chapter 4). 

a probability sample of clients rather than all clients would be appropriate 
for this project. Similarly, projects that encounter cost or time constraints 
may need to consider utilizing a sample of clients in lieu of a ccnplete 
census of clients. Personnel collecting the data should be trained to select 
the sanple in accordance with a pre-deteimined sampling protocol (in nost 
cases a siitple randan sanple will suffice) and properly administer the instru- 
ments in order to ensure that the collected data are reliable and valid. 



CHAPTER 2 
FOCUSING THE EVALUATION 



Evaluations that are most useful and meaningful have direction and 
purpose. They reflect thoughtful, systematic planning that insures respon- 
siveness to constituencies and has implications for future program 
decision-making. 

Secondary/transition projects in particular should have well-designed, 
focused evaluation plans. The lack of knowledge among the general public 
about these projects and the need for effective, replicable program designs 
for agencies and institutions attempting to inplement secondary/transition 
projects make it incumbent upon those projects already in place to share 
their successes and shortcomings. Only in this way can advances be made in 
enriching the lives and expanding the opportunities of individuals with 
handicaps. Through the dissemination of well-documented project evalua- 
tions that substantiate effective practices , potential employers, sponsor- 
ing agencies, advocates and handicapped youth/adults (and their families) 
can be alerted to new ideas that enhance businesses, conmunities, and 
individual lives. 

This chapter addresses five steps which should be taken to focus a 
secondary/transition evaluation. They are: 



Step 1 



Step 2 



Step 3 



Step 4 



Step 5 



Describing Evaluation 
Setting 



Identifying Evaluation 
Audiences 



Stating Evaluation 
Rationale 



Determine Evaluation 
Questions 



Identify Evaluation 
Constraints 
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The process begins by describing the setting (Step 1) of the specific 
transition project. By setting, we mean the context in which the project 
exiscs and functions. It consists of the environment, plus project 
purpose, operational procedures and resources, with the description of the 
evaluation setting, it is sound evaluation planning to identify the 
audience (Step 2) c I Involve those audiences most directly affected by the 
evaluation results, in this way, they become "stakeholders" and will be 
more likely to understand and use the evaluation results. Once a compre- 
hensive picture of the project and its setting is in place, and the 
audiences have been identified you can determine the rationale (step 3) for 
the evaluation. The rationale determines in large part which evaluation 
methodology you should use. The concerns of the stakeholders and the 
project personnel should determine potential evaluation questions (Step ■*). 
The determination of the questions should be relatively straightforward. 
At this point evaluation constraints (step 5) should be identified as well 
as their likely impact upon the evaluation. The principal constraints deal 
with such factors as budget, timeframe, agency regulations, and political 
considerations . 

A. Step 1: Describing the Era 1 nat ion Setting 

Planning to evaluate a secondary/transition project should begin with a 
complete description of the project. The description clarifies project 
rationale, goals and objectives; specific project components and opera- 
tions; explains personnel roles and responsibilities, project resources; 
and describes the context of the project within the host organization, 
system and/or community. 

1. Project Rationale 

The project rationale explains the controlling, underlying princi- 
ples on which the project is based. Put simply, it answers the questions 
as to why the project, was undertaken. 
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PROJECT RATIONALE 

In Minneapolis a postsecondary project is 
based on the belief that learning disabled 
adults in mainstream postsecondary settings 
need long-term modificatioiis to their writing 
process that enable them to successful ly meet 
the writing demands in both academic ^-d occu- 
pational settings. Typically, however, "stop- 
gap H measures of peer tutors or remedial lab 
workers are all that is provided to assist 
them catplete the writing courses and/or 
writing assignments required to successfully 
catplete a degree or occupational certificate 
program. Such intervention is not likely to 
assist learning disabled adults acquire the 
skills to produce clear, concise and coherent 
written pieces. On the other hand, anecdotal 
information in the literature and recent but 
limited intervention in the General College of 
the university of Minnesota have 3hown that 
using a multi-media curriculum in writing is a 
very premising approach to help students 
acquire the much-needed skills. 



2. Project Goals 

Project goals extend the rationale and reveal what a project is 
trying to accomplish. Goals are best stated in terms of general desired 
outcomes 



PROJECT GOALS 

The Minneapolis project has as its over- 
all goal "the development, testing, inplemen- 
tation, evaluation, and dissemination of 
varied-media curricula in writing for main- 
streamed learning disabled postsecondary 
students, with transition-to-work orienta- 
tion." The curriculum is to be generic and 
easily replicable by other academic 
institutions. 



3. Project Ob jectives 

Project objectives add concrete detail to project goals by stating 
what specific outcomes are desired. 



PROJECT OBJECTIVES 

The specific, measurable objectives in 
the Minneapolis goal are: 

• Increase the retention and academic 
performance levels of mainstreamed 
learning disabled postsecondary 
students. 

• Increase the ability of learning 
disabled postsecondary students to 
use varied media, including word 
processing, to cxxmainicate effec- 
tively in writing. 

• Increase awareness of cannunication- 
related enploymant adaptations 
available to learning disabled 
persons to facilitate transition-to- 
work. 

• Disseminate knowledge and findings 
in a timely, effective way with the 
goal of supporting replication and 
adaptations to other settings. 



4- Project Operatinnfi/flr ±iviti efi 

Project operations/activities are those formal and informal proce- 
dures and rules used in a project. They describe project implementation by 
projecc staff: who does what in the project, and how they do it. A list 
of project operations/activities could include: 

• Client selection and assessment procedures, 

• Service delivery procedures, 

• Employer contact procedures, 

• Formal and informal project rules and norms, 

• Project planning and management, 

• Personnel management, and 

• Project financial management procedures. 
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PROJECT OPE3»TIONS/^TIVITIES 

The eight key activities of the 
Minneapolis project are the: 

(1) Creation of a research base in 
writing and computers, composition 
for ID adult students, and related 
areas; 

(2) Translation of research findings 
into a curriculum plan for varied- 
media offerings in a writing course; 

(3) Identification of 20-24 ID students 
each year to participate in the 
project; 

(4) Implementation of the intervention 
writing syllabus in both courses; 

(5) Enrollment of participants in a 
career planning course, during which 
their interests/aptitudes are 
assessed, career networking is done, 
and a career/ goals/academic 
planning contract is drawn up; 

(6) Evaluation of project activities; 

(7) Dissemination of evaluation results; 

(8) Planning is done for the following 
year based on research and evalua- 
tion results. 



5. Project Resources 

In addition to operations/procedures, projects use various 
resources in pursuing tneir goals and objectives. Resources directly 
linked to project success are an inportant part of the project description, 
such as: 

• staff qualifications, 

• funding sources and amounts, and 

• conmunity resources available to the project. 

Project resources are mechanisms to achieve project goals and objec- 
tives, it is inportant to identify resources essential to project success. 
A list of resources would include the following: 

• project staff and their responsibilities, 

• physical environment/site, and 

• equipment, materials, and supplies. 

A sanple description of one of the project resources is presented below. 
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PROJECT RESOURCES: STAFF 

The staff of the Minneapolis project 
consists of a director/lead teacher/ dissemin- 
ator, an ID consultant specialist, an 
evaluator, a research assistant, a secretarial 
assistant, and tutors. The director /lead 
teacher /disseminator has demonstrated ability 
as a teacher of writing and is well-acquainted 
with adoptive media for LD adult learners. 
The ID consultant's background is special 
education and knowledge of the employment 
needs and difficulties of ID adults. 



6. Project Contgyfr 

The final and very important part of the project description is 
the project context, it is defined as the setting of the project, such as 
whether it is located in a university or in a private agency. The context 
often determines how the project is set up and how it operates, services 
provided by a private agency, for exanple, may be inappropriate within a 
university. A context description could include the following: 

• Other organizations and agencies serving the handicapped and 
jobless, and their relationship to the transition project, 

• The local economy, especially types of industries and employ- 
ment opportunities, 

• Unemployment rates and trends, 

• Public transportation, and 

• Community support for the project. 
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PROJECT CONTEXT 

The Minneapolis project is located in the 
General College of the University of 
Minnesota; the College is a 50-year-old open 
admissions college within this major research 
university. The College has a history of 
offering programs and curriculum that have 
served as models for many cannunity colleges 
and postsecondary institutions, It is a test- 
ing ground for undergraduate lower division 
curriculum and also has a history of testing 
and disseminating curriculum for "bypassed 
populations," e.g. learning disabled students. 
The number of learning disabled persons on the 
canpus of the University of Minnesota alone is 
growing, frcm 14 in 1978-79 to 79 in the fall 
quarter of 1985 - a 560% increase. All these 
students are in programs that require either 
the successful completion of writing courses 
or demonstration of writing proficiency. 
Additionally, the workplace is increasingly 
cximsunication-intensive , according to the 
Department of Labor's Occupational Outlook 
Handbook 



The inportance of a cotplete description of the project cannot be over- 
emphasized, it is needed to plan any evaluation of the project. It is 
through the project description that the reader fully understands the 
project in operation. It also enables replication of successful projects 
elsewhere. 
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B. Step 2: Identifying Evaluation Audiences 

A complaint leveled against many evaluations is that they ask questions 
nobody cares about and provide answers nobody can use. This situation can 
be minimized by ensuring, from the start of the evaluation, that those 
individuals having a direct interest in the project—the project stake- 
holders—participate in the evaluation. This can be done by determining 
what information stakeholders want from the evaluation (and when they want 
it) . At this point a sense of reality and balance must enter into the 
evaluation plan. It is inportant not to attenpt to satisfy the information 
needs of more audiences than the project can afford in terms of resources 
and time. Stakeholders also can serve as important reviewers of the 
evaluation methodology, offering advice on topics such as prioritizing 
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proposed evaluation questions and how best to frame than. Stakeholders my 
also review data analysis plans, be involved in discussions of preliminary 
evaluation findings, and discuss reporting techniques. 

Several groups and individuals could be included in a project's evalua- 
tion audience: 

• Project clients, 

• Parents/guardians of project clients, 

• Agencies participating in the project, 

• Project staff, 

• School staff, 

• Employers, 

• Advisory group personnel, 

• Sponsoring agency personnel, 

• Iccal decisionmakers (e.g., vocational education director, 
rehabilitation services agency director, school 
superintendent), 

• Project content specialists (e.g., curriculum personnel, 
evaluation staff), and 

• Others (e.g., legislators, potential adoptees). 

In addition to information needs, the evaluator should be aware of the 
project roles and responsibilities, and biases, of stakeholders. This will 
help decide how much effort should be devoted to satisfying those needs. 
Priorities have to be set. Information needs of stakeholders far removed 
from direct project operations, for exanple, may be assigned a lower 
priority than the needs expressed by those stakeholders responsible for 
service delivery. The latter group probably would make more iitmediate use 
of the evaluation results. Higher priority stakeholders may have needs 
addressed throughout the project duration while lower priority audiences 
may receive information during a second or third year of the project 
period. If at all possible, some consistent flow of information to all 
stakeholders is most desirable. A monthly update or quarterly newsletter 
may be vehicles for this conmunication. 
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PROJECT STAKEHOLDERS 

The Setting 

In Fort Collins, Colorado, a project 
serves moderately-severely disabled persons 
who are unenplqyed, under-enployed , or "at- 
risk - for enployment. The project provides a 
combination of interagency and ongoing 
enployer linkages and intervention with the 
target youth and adults. Aiditionally, the 
project (located at Colorado state University) 
will train students as vocational evaluators 
and trainees. 

The Stakeholders 

The audiences/stakeholders identified by 
staff at the project are: OSERS staff, state 
legislators, state agency staff, staff of 
potential funding sources (including state 
agencies), university faculty and students, 
staff at potential replication sites, the 
project Advisory Council, project administra- 
tors and staff, public school staff, 
enplqyers, consumers, and parents. 

The many stakeholders identified clearly 
have different information needs that are 
specified below. OSERS staff is a stakeholder 
group that */ill receive nearly all information 
about the evaluation. On the other hand, 
university faculty and students at Colorado 
State university are the target audience on * 
evaluation information and data about pre- 
service training of graduates and under- 
graduates to assure direct service roles in 
community-based vocational programs for 
persons with disabilities. Consumers and 
their families are to receive evaluation 
information about the development of comnunity 
employment and training opportunities, the 
establishment of linkages between project 
staff and existing service agencies, the 
nuntoer/proportion of consumers placed in mean- 
ingful cannunity employment, and the extent to 
which consumers have been trained and 
supported (measured by consumer /employer 
satisfaction) . Consumers and their families 
may not receive evaluation information on the 
preparation of the CSU students, on project 
dissemination efforts out of state, the estab- 
lishment of service linkages, or project 
influence on CSU course content. 
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C £t£P_3j — Determinin g the Evaluation Rational* 

There are numerous reasons for evaluating secondary/transition inter- 
vention projects. The project director may want to know, for example, if 
project services are reaching those who are most needy or if activities 
that appear to be of minimum effectiveness should be continued. Regardless 
of the reason, :t is important that everyone involved directly in the 
evaluation share an understanding of why the project is being evaluated. 
An evaluation may be a single or multiple purpose study. Some of the 
reasons for evaluating a transition intervention project include: 

• Meet the funding agency's requirements for an evaluation, 

• Identify the characteristics of those persons who are 
receiving services, 

• Measure project impact and cost-effectiveness, 

• Produce information useful for project replication, 

• Identify project improvement initiatives, 

• Reveal any project side-effects, 

• Uncover any unmet needs, and 

• Aid the improvement of adaptive equipment. 

The evaluation rationale will determine in large part the appropriate 
methodology to use. For example, project management can be evaluated using 
a management by objectives schema to ensure that project tasks are being 
completed on-time and on-budget. In determining the rationale for your 
project evaluation, you also should determine what types of information are 
needed/desired by the stakeholders (see Step 2), for what reasons, and at 
what time. The information needs of the project's stakeholders often shape 
the rationale. 
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EVALUATION RATIONALE 
The Setting 

In Blocraington, Indiana, Project COMPETE 
is a cooperative project that is designed to 
develop and inplement a cooperative and coor- 
dinated approach to effectively train youths 
aged 16-22 with severe handicaps so that they 
can be expected to maintain competitive 
employment. This project focuses on estab- 
lishing formalized linkages among a public 
school system, a cxmounity-based rehabilita- 
tion center, and other caimunity-based 
agencies to insure an integrated continuum of 
services that enable severely handicapped 
youths ';o obtain and keep ccnpetitive employ- 
ment. The role of the school system will be 
to provide vocational exploration, training in 
work habits/attitudes, work experience, and 
training in clusters of related skills; the 
role of the rehabilitation agency will be to 
provide job-related training, support services 
when clients are placed in jobs, and to 
identify/coordinate services to be provided by 
other agencies. The project is designed to 
demonstrate how rehabilitation centers can 
serve as transition agents in this system, to 
validate/replicate processes that enable such 
agencies and the public school system to form 
linkages, and to validate a staff development 
program that provides interdisciplinary train- 
ing for both rehabilitation facility and 
public school staff so the project model can 
be successfully implemented. 

The Evaluation Rationale 

The purpose/rationale of the evaluation 
is to answer two interrelated questions: 
(1) does the levelopment and implementation of 
the cannunity-based rehabilitation model meet 
project objectives, and (2) does the rehabili- 
tation center model implemented at each field 
site address the needs of local clients and 
service providers? In order to answer these 
questions, the evaluation will assess the 
project's adequacy in addressing the needs of 
its clients (unmet needs; project replication 
information), the relation between intended 
and actual implementation (project improve- 
ment), the impact on clients and goals 
attained by the project, and the project's 
efficiency in terms of costs and benefits. 
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D. Step 4; — Determining Evaluat ion Questions 

Once a thorough project description is developed, >the evaluation 
audiences are identified, and the evaluation rationale is established, the 
step of determining evaluation guestions is relatively straightforward. 

The information needs of the project's stakeholders largely should 
determine the ^valuation guestions, since the evaluation should respond to 
those needs. In addition, project evaluators may be interested in collect- 
ing other information to answer guestions about program components . They 
may be interested in applying a new evaluation technique (e.g. , interview 
guide) to find out if it works well in the project setting. Another source 
of evaluation guestions might be current theory about effective transition 
intervention practices. 

Each evaluation question should be: 

• Relevant (provide meaningful information for the project 
being evaluated) , 

• Measurable (there must be an identified mechanism for obtain- 
ing the information so as to obtain a meaningful answer to 
the question of interest), and 

• Achievable (the question must be answerable within the 
resources and timing available for the evaluation) . 

As stated earlier, evaluation questions should be limited to a manageable 
number. To assist in the final selection of evaluation guestions, apply 
the above criteria to each question and then rank order the evaluation 
questions in order of importance. Determine the evaluation audiences for 
each question and the potential impact of the data obtained in response to 
each question. Then, select as many of the guestions as are manageable. 

The development of evaluation criteria for each guestion should further 
refine the evaluation guestions. These criteria should specify the level 
of expected performance and/or due date for each evaluation guestion. For 
example, if there is to be a student selection process, the expected date 
for completion of this process and the anticipated number of students to be 
screened and selected should be specified. Likewise, if client placements 
are the focus of an evaluation guestion, the number of students to be 
placed and anticipated placement dates (e.g., within 90 days of program 
entry; within 15 days of skills training) should be specified. 
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EVALUATION QUESTIONS AND CRITERIA 

Tlie staff at the Fort Collins project 
identified 5 formative evaluation questions, 
i.e., questions related to project monitor- 
ing/iitprovenent, and 4 suntnative evaluation 
questions, i.e., related to pro^jct outcomes. 
Ttoo of these questions, audiences, criteria, 
responsible staff person, and ccqpleKon date 
are in jhe matrix below. 





EVALUATION QUESTION 


1. What gaps exist in voca- 
tional services for 
persons with disabilities 
in the Fort Collins area? 


? . How many and what pro- 
portion of consumers 
have been placed into 
meaningful comnunity 
enplqyment? 


STAKEHOLDERS/ 
AUDIENCES 


OSERS 

Replication Sites 
Various State Agencies 
(Dept. of Education; 
Division of Rehabili- 
tation; Division of 
Developmental 
Disabilities 


OSERS 
Consumers 
Families 
Advisory Panel 
Replication sites 
State Agencies 
Local/State Funding 
Sources 


CRITERIA 


Expressed need documented 
by project staff from 
specific sources (ARC, 
consumers, rehabilita- 
tion center) 


70% year 1 

consumers still 

employed as cf 8-30-88 
80% year 2 

consumers still 

enployed as of 8-30-88 
90% of year 3 

consumers still 

enployed as of 8-30-88 


STAFF PERSON 
RESPONSIBLE 


Project Director 


External Evaluator 


DA1E OF 
CTJPLETION 


February 1, 1986 


August 30, 1988 


DATA 

COLLECTION 
STRATEGY 


Meetings/discussions with 
local service systems, 
families, and consumers; 
documentation of needs 
expressed. 


Follow-up with con- 
sumers through direct 
contact, telephone 
interview, employer 
interview. 
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An important part of the evaluation question screening process involves 
considerations of the constraints that may limit the evaluation. 



E. Step 5: Identifying Evaluation Constraints 

As indicated, planning an evaluation should take into account 
constraints that can affect the evaluation. Identifying the relevant con- 
straints «nd assessing their likely impact upon the evaluation are key 
steps in determining which evaluation approach is feasible. Several poten- 
tial constraints are listed below: 

• Agency Regulations: several agencies may be involved in the 
evaluation process, each with their own guidelines or proce- 
dures for conducting evaluations and reporting results; 

• Protection of Human Subjects: moral, ethical and legal 
guidelines for protection of the rights of human subjects 
should be followed; 

• General evaluation standards: criteria for a valid evalua- 
tion methodology including valid measurement, systematic data 
control, appropriate data analysis procedures, justifiable 
conclusions, and objective reporting are followed; 

• JDRP Evaluation standards: specific criteria set by the 
Joint Dissemination Review Panel to determine exemplary 
project status are observed if JDRP approval is sought; 

• Personnel: availability and evaluation skills of project 
staff; 

• Resources: availability of resources earmarked for project 
evaluation; 

• Timing constraints: availability of time to complete the 
evaluation; and 

• Political constraints: any limits either on the evaluation's 
questions, or on the methods that can be used to anewer them. 

Sometimes constraints do not arise until the evaluation effort is well 
underway, and political constraints may arise during reporting time. Every 
attempt shou'.d be made to anticipate evaluation constraints and address 
them early on; however, if constraints do occur near the reporting stage of 
the project, every attempt should be made to ensure an open, honest evalua- 
tion and reporting effort. Third-party evaluators are seen as one method 



for encouraging this "freedom" in evaluation, although objective internal 
evaluators can be expected to conduct fair evaluations. 

Appendix B provides conpleted worksheets for the examples cited in this 
chapter as an assist to the reader in carrying out the processes required 
to systematically focus the evaluation. 
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CHAPTER 3 

PLANNING AND IMPLEMENTING THE EVALUATION 



The actual evaluation of a project is but one part of the total program 
effort, and should be planned so that disruption to the ongoing program is 
kept to a ininiinum. The procedures that are planned should be realistic, 
given the constraints of resources available and needs of stakeholders. If 
there are circumstances that are likely to interfere with the collection of 
valid, reliable data using a particular instrument, alternative methods 
should be considered and attempts should be made to alter the circumstances 
or the instrument. Attention to assessment instruments is particularly 
critical for secondary/transition projects because many instruments 
ccratonly used in the general population may not t j suitable for handicapped 
persons. Chapter 3 will discuss evaluation approaches, management plans, 
and data collection/analysis procedures (Figure 1) with these parameters in 
mind. 
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A. Determining Evaluation A pproach /Method 

There are a large number of approaches to evaluation and it is beyond 
the scope of th:\s document to discuss *J. . of them. Many evaluations can be 
characterized as either formative or sunroative evaluations as originally 
proposed by Scriven (1967). Formative ewtinatu' i is concerned with deter- 
mining the worth of a project while in process so that it can be modified 
or revised as needed. In addition, the details of formative evaluation can 
be extremely valuable for replication purposes. Suirmative evaluation is 
concerned with determining the overall worth of a project after it has been 
sufficiently developed. Many of the same data collection methods, instru- 
ments, and data analysis techniques can be used for both types of evalua- 
tions. Often, only the purposes of the evaluation (e.g., program improve- 
ment versus replication), the timing of the evaluation (e.g., early on 
versus near project cotrpletion) and the audiences for reporting (e.g., 
project staff versus external decision-makers) differ between formative and 
suirmative f valuation efforts. 

Most frequently, formative evaluation focuses on describing the actual 
treatment and the implementation of the objectives, processes, and compon- 
ents of the project in the early stages of project activity. It is a 
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process analysis of the organization and service delivery components of the 
project. Formative evaluation is concerned with whether the project was 
properly implemented with respect to, for example, services provided to 
transition project clients. Examples of formative evaluation questions 
are: Are there discrepancies between the expenditures and the project 
budget? Has the project been staffed appropriately? This type of evalua- 
tion needs input from the internal project staff who are familiar with the 
day-to-day operations of the project, and may even be conducted by project 
staff. 

Sunmative evaluation continues where formative evaluation ends. It is 
usually concerned with the impact or effects resulting fx<xa. project 
services or activities. Examples of sunmative evaluation questions are: 
How much did the project improve the independent living skills of handi- 
capped clients? How many more clients gained competitive atployment 
positions as a result of the project? Sunmative evaluations generally 
involve more rigor than formative evaluations since you need to make sure 
any impacts discovered are a result of the project and not due to outside 
influences such as favorable change in local economic conditions. This 
type of evaluation generally is conducted by an independent or external 
evaluator. 

Both evaluation approaches are important and complement one another. 
For example, a sunmative evaluation may indicate no impact on project 
clients. The formative evaluation may indicate that some service delivery 
con*, jnents of the project were not properly implemented and thus suggest a 
reason for the lack of project impact. 

In some instances, projects elect to conduct both formative and sunma- 
tive evaluations. Beyond providing impact information for external 
audiences, they also want to analyze what has/has not worked in the 
delivery of the program with the goal of making decisions to improve the 
program. In addition, projects may want to document their ongoing activi- 
ties in order to initiate similar programs throughout their comnunities to 
other agencies or for broader dissemination. As implied in Chapter 1, 
there is a need to "sell" the public on the value of programs that work. 
Formative/summative evaluation facilitates this. 
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In a multi-year project, for example, a formative evaluation may be 
conducted during each year of the project. The formative measures may be 
used to: 

• Provide evidence of project implementation, 

• Determine discrepancies between project operations and 
project plans, and 

• Partially satisfy evaluation requirements of the funding 
agency. 

The target audiences for this portion of the evaluation may include: 

• Office of Special Education and Rehabilitative Services, U.S. 
Department of Education, 

• Handicapped education association personnel, 

• Special education personnel, 

• Rehabilitative services personnel, 

• Vocational education personnel, 

• Project staff, 

• Clients, and 

• Client parents/guardians. 

The focus may be on project improvement in the early stages and on 
project refinement in the later stages . The evaluation may be conducted 
internally under the direction of the Project Coordinator and may be 
augmented by an external evaluator each year. 

The second ccnponent of the evaluation approach is generally a sunma- 
xlve evaluation conducted by an external evaluator during the third year of 
the project. The focus of the summative evaluation would be estimating 
project impact. The evaluation results also can be used to: 

• Produce data to support continued project funding at the 
local level, and 

• Partially satisfy evaluation requirements of the funding 
agency. 

The audiences to be addressed by the results of this sunmative evaluation 
may include: 

• Office of Special Education and Rehabilitative Services, U.S. 
Department of Education, 

• I£A (local education agency) policymakers, 

• Clients, and 

• Future client parents /guardians. 

25 
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Additional evaluation approaches are discussed in several of the 
annotated bibliographies found in Appendix A. 

B. Developing an Evaluation Management Plan 

An evaluation management plan is a tool which can be used to monitor 
the conduct of evaluation tasks and activities. Much of the information 
for this plan will be available from the previous steps in the evaluation 
process (e.g., identification of projec- objectives and related evaluation 
questions) . other information will have to be gathered and incorporated 
into the plan (e.g., evaluation activities associated with each evaluation 
question). The evaluation plan should contain the following: 

• Project objectives (e.g., place clients in competitive 
employment), 

• Evaluation questions linked to each project objective (e.g., 
to what degree have clients been successfully placed iu 
competitive employment?), 

• Evaluation activities related to each evaluation question 
(e.g., summarize the match of clients' interest and skills 
with the job requirements ) , 

• Staff responsibilities for each evaluation activity (e.g., 
project coordinator and evaluator) , 

• Completion dates (milestones) for each evaluation activity, 

• Data sources for each evaluation activity (e.g., placement 
records, job skill analysis forms, and job analysis forms), 
and 

• Criteria for the successful completion of each evaluation 
activity (e.g., proportion of clients placed in jobs which 
match their top five skill and interest areas). 

The above list of information would be provided for each project 
objective. 

Gantt Charts could also be developed to indicate the functional and 
time interrelationships among the evaluation activities. When properly 
developed these charts indicate the extent to which the initiation of a 
particular evaluation activity was dependent upon the successful completion 
of one or more other evaluation activities. 
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The Gantt Chart is perhaps the most commonly used time-oriented control 
device. It not only provides the developer of the evaluation with a tool 
for viewing the overall schedules, including reporting obligations, but, of 
equal importance, the chart also provides standards a gain st which actual 
achievements can be ccnpared as work progresses. An example of a generic 
Gantt Chart is provided in Figure 2. The necessary requirements for 
construction of a Gantt Chart are: 

• Identification of subtasks 

• Numbering (coding) of subtasks 

• Logical scheduling of subtasks 

These requirements should flow easily frai> a well developed work plan. The 
chart contributes directly to the control function when the variable of 
concern is time/ schedule. Indirect contribution is made by Gantt Charts to 
the control of cost/personnel utilization. 

In sumnary, the evaluation management plan is a tool to help ensure 
that each evaluation question is properly addressed by specified evaluation 
activities carried out by specified staff within a specified tide frame. 



Figure 2 
Generic Gantt Chart 



Tasks and Subtask 



Task 1: Focusing the Evaluation 

1 . 1 Describing Evaluation 
Settings 

1.2 Identifying Evaluation 
Audiences 

1.3 Stating Evaluation 
Rationale 

1 . 4 Determining Evaluation 
Questions 

1.5 Identifying Evaluation 
Constraints 

Task 2: Planning and Implement- 
ing the Evaluation 

2 • 1 Determining Evaluation 
Approaches/Methc Js 

2.2 Collecting Data 

2.3 Analyzing Data 

Task 3: Reporting and Using 

the Evaluation Results 

3.1 Reporting Evaluation 
Findings 

3.2 Using Evaluation 
Findings 
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EVALUATION: MANAGEMENT 



The Fort Collins project developed the 
following evaluation management plan for e 
formative evaluation question. 

TRANSITION TO COtMJNITY EMPLOYMENT 
PQRMKTIVE EVALUATION 
TASK 5.0 

TASK: Project dissemination within 
western region of U.S. 

RESPONSIBLE PERSONS: Project director, 
projsct consultants 

METHOD FOR ASSESSMENT: Wbrkshqp evalua- 
tion forms and participant questionnaire. 

CRITERIA FOR COMPLETION: An average 
rating by workshop participants of 4.0+ (on a 
5 point scale). 

AUDIENCES: OSERS, direct service and 
administrative personnel frcm public school 
and postsecondary vocational programs, univer- 
sity faculty front human service fields, repre- 
sentatives from state education, vocational, 
and rehabilitation systems. 

DISSEMINATION METHOD: Third year work- 
shop and accotpanying written materials. 

DUE DATE: Sumner, 1988 

SUB TASKS: 

5.1 Document project components for 
publication and dissemination. 

5.2 Wbrkshqp promotion. 

5.3 Wbrkshop logistical 
preparations. 

5.4 Establish linkages with poten- 
tial replication sites for on-going 
interaction and consultation. 
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C. Planning and Erarntin q Data Collection PrnraHnres 

The resources available to evaluate most transition projects are rather 
limited and hence extensive data collection activities are, in most 
instances, infeasible. Consequently, the evaluation strategy should 
concern itself with collecting readily available data on project implemen- 
tation and effectiveness to the extent feasible to adequately gauge the 
success of the transition project. Administrative records are readily 
available. Interviews, checklists, rating forms, and short questionnaires 
provide other easy and economical ways to collect data for either a forma- 
tive or sunmative evaluation. 
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'Types 04. administrative data and other information that the project 
will generate should be specified at the outset as part of its normal 
record-keeping. Linking these data to the goals and the objectives of the 
project will ensure that the project evaluators will have at least some 
relevant data to work with in carrying out a transition project evaluation. 

Characteristics of good data collection procedures are: 

• Flexible and adaptable, 

• Unobtrusivenes s and appropriateness (e.g., proper reading 
level) for the intended audience, 

• Reliability (accurate measurement), and 

• Validity (measuring what you intended to measure) . 

The data collection procedures should be developed so that each data 
element has a specific use in the evaluation plan. InformaU.on for which 
there is no advance use should not be collected. 

Most transition projects are not so large as to require sampling of 
clients; however, there are exceptions. The North Dakota project currently 
has in excess of 2,000 clients. Surveying all clients in this project 
would be neither necessary nor cost effective. Hence, collecting data from 
a probability sample of clients rather than all clients would be appropriate 
for this project, similarly, projects that encounter cost or time constraints 
may need to consider utilizing a sample of clients in lieu of a complete 
census of clients. Personnel collecting the data should be trained to select 
the sample in accordance with a pre-determined sampling protocol (in most 
cases a simple random sample will suffice) and properly administer the 
instruments in order to ensure that the collected data are reliable and valid. 

The data collection activities must be closely monitored in order to 
determine that proper procedures are being followed and to ensure that the 
data are being collected in a timely and cost effective manner. In 
particular, the response rates should be high enough to guard against bias 
due to non-responae. One technique which is frequently employed when 
response rates are unacceptably low (say less than 75%) is to exert a 
special effort to secure, data from a sample of non-respondents and deter- 
mine whether they differ substantially from the respondents. The data 
instruments should be carefully checked for missing items and inconsistent 
responses among items. If the missing items are critical with respect to 
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the evaluation plan, then it may be necessary to go back to the data source 
(e.g. , the client) for the missing information. Critical inconsistencies 
could also be resolved in this manner. Other inconsistencies may be 
resolved by logical editing rules. Data entered into a computer file 
should at least be partially verified to reduce data entry errors. 
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DATA COLLECTION: STRATEGY 



In the Indiana project, the formative 
evaluation ccnponent will examine the adequacy 
of the project in meeting client needs and the 
relation between intended and actual project 
operations; the sunmative ccnponent will 
examine project iitpact and efficiency. All 
data listed below will be readily available; 
much of it is available from existing project 
records. 

Data regarding adequacy will consist of 
semi-structured interviews with staff regard- 
ing perceptions of training and opinions of 
changes/modifications . 

Data regarding relation between action/ 
inte n ded operation will consist of observa- 
tions of changes in learner/ instructor 
behaviors; structured observation regarding 
classroom compos ition/climate/activities ; 
standard and criterion-referenced measures of 
increases in instructor/ client knowledge; 
ratings of client performance (work 
attitudes/behaviors /work and interpersonal 
skills); document analysis of IEPs and IHPs. 

Data regarding iitpact on clients will 
consist of interviews with clients/parents/ 
employers regarding satisfaction; interviews ' 
with agency personnel regarding inservice 
training and linkages and methods for main- 
taining/expanding linkages; appropriateness of 
linkages; measures of satisfaction with 
program operation; collection and analysis of 
demographic data about number of placements in 
ccnpetitive employment; number of clients 
placed; clients' handicaps; number of clients 
still in ccnpetitive employment at end of 
year; nature of work in each placement; hours 
worked per client; wages earned per client; 
number of job terminations; and reasons for 
job terminations. 

Data regarding cost benefit analysis will 
consist of program costs (number of training 
hours per client; fiscal expenditures asso- 
ciated with training hours; number of training 
hours per client on job site; fiscal expendi- 
tures associated with training on job site); 
length of on-site support; number of follow-up 
contacts; and fiscal expenditures associated 
with on-site support and follow-up program 
benefits (wages earned, taxes paid, savings in 
SSI benefits from placement in ccnpetitive 
employment) . 
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D. Analyzing Data 

Many evaluation questions can be addressed by a few sinple descriptive 
statistics. This is especially true of formative evaluations where the 
transition program ccnponents and services received by the program clients 
are being described. For exanple, the client population can be described 
in terms of average age, average education level, and average aptitude or 
achievement score levels. Likewise, the program ccnponents can be 
described in terms of the number of clients exposed to various transition 
program components and the frequency with which they are being served by 
those components. 

Data analyses associated with suitmative evaluations are nore complex 
since you are trying to measure the iitpact of the project on client out- 
comes. The analyses must rule out carpeting explanations (e.g., improved 
local economic conditions) if we are to attribute iitproved client outcomes 
to the transition project. The best way to estimate the effect of a tran- 
sition project is to randomly assign potential clients to the transition 
project and to a "no- treatment 11 control group, and then compare the two 
groups on outcome measures (e.g., number gainfully enployed) after project 
clients complete the program. This usually is not possible because of 
practical or ethical considerations. The next best nethod is to collect 
data on client outcome measures before and after exposure to the transition 
project and ccnpare the client changes to changes for a similar available 
(comparison) group that was not exposed to the transition project. These 
are called nonequivalent control group designs. If data on a similar group 
are not available, descriptive needs data may exist from local /state/ 
national assessments which can be used for comparison purposes. 

There are statistical techniques such as regression analysis that allow 
for the estimation of transition project impacts by comparing the outcomes 
for the transition project clients and the nonequivalent control group 
members. There are a number of technicalities that must be considered in 
these catparative analyses. For exanple, the sanple sizes must be large 
enough to consistently detect a reasonably sized inpact of the transition 
project. Also, if there are known differences between the transition 
project clients and the control, or catparison, group members, then these 
differences must be measured and incorporated into the regression analyses 
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so that the transition project impact estimate can be adjusted for those 
differences. If the data warrant conplex analyses, and the evaluation 
staff are not firmly grounded in these techniques, the assistance of an 
expert consultant should be considered. 

Microcomputers can be useful in analyzing the collected data. Software 
is available to generate the descriptive statistics associated with a 
formative evaluation and the regression analyses associated with a sunroa- 
tive evaluation. 

References for basic analysis procedures are given in Appendix A. 
Worksheets for Chapter 3 may be found in Appendix C. 
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CHAPTER 4 

REPORTING AND USING THE EVALUATION RESULTS 



A carefully planned, meticulously executed evaluation will, throughout 
its implementation, have an underlying purpose - that of presenting clear, 
interpretable program results to stakeholders. The early-on identification 
of evaluation audiences, their roles, biases, and information needs will 
facilitate a cogent presentation of the results. The forms in which 
results are presented will reflect thoughtful consideration of the 
audiences, and the outcomes that presenters hope to achieve through the 
dissemination of information to each audience. Desirable outcomes may 
include: 

• Increased (or at least sustained) investment in the program, 

• Increased effectiveness and efficiency within the program, 

• Increased visibility of the program to motivate replication 
beyond the inmediate site or efforts, 

• Greater understanding on the part of parents and/or program 
participants, and 

• A demonstration of quality responsiveness to requirements for 
accountability . 

In this chapter we will discuss the plann ing and reporting of evalua- 
tion results, and how those results may be used by program planners, 
supporters, and other interested parties. A related worksheet is presented 
in Appendix D. 



A. Planning Presentation of BSSUltS 

In order to insure clearly stated, well-organized evaluation reports, a 
planning stage should precede the actual writing of the report. Once 
critical laments of the report are identified, outlined, and review by a 
team of program representatives, the actual writing can begin. Planning 
for the presentation of results should include 

1- Defining the Purpos e of Your Evaluation for Your Audiences 

Your evaluation questions facilitate this, with a description of 
how your data collection and sunmary methods answer those questions. 

2. Identifying Audiences 

Identify those audiences who will be interested in your findings, 
and those audiences you will want to interest in your results. Details of 
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the report, kinds of graphics and charts displayed, and language level are 
likely to vary according to the needs of the audiences. Major decision 
makers are likely to want more details than others. For example, program 
administrators and supervisors need to know if project goals and objectives 
have been net, and a clear description of program processes/procedures. 
Those providing program funding will likely require the most detail in both 
the actual report and in the executive summary. Curriculum specialists 
need to know if materials and methods have been appropriate to meet goals 
and objectives. Advisory boards want to know how well the program is func- 
tioning, its areas of strength and weakness, what is necessary to maintain 
the strengths and improve the areas of weakness. The news media should 
receive accurate, easy to understand o>*a displays with appropriate inter- 
pretations based on the presented data. Participants/ students need to 
know their progress in meeting individual objectives. 

3 - Se l ecting tr.e Technignp for Pr^nM"^ Y ^i r ft fltr i 
Descriptive data include average responses, range of responses, 

frequency distributions, percent of responses in different categories, 
correlation coefficients to show the degree of relationship between two 
variables, and summaries of narrative data from interviews, questionnaires 
or anecdotal records. Comparative data include such statistical methods as 
chi-square, t-tests, analysis of variance, and multiple regression. Most 
importantly, the data must be presented so that the audience can understand 
it. Cross-check any subjective results with records, if possible, and 
through discussion with appropriate others involved in the program. This 
can lend credibility to the conclusions and help ensure that recommenda- 
tions will be accepted. 

4 - Id ent i fying the Method t*\ y«i win u se te caaximisate Evalua- 
tion Results 

Different audiences are able to use different types of reporting 
more effectively. Written reports, oral reports, reports primarily graphic 
in nature, slide/tape presentations, and combinations of the foregoing are 
all likely to be used, depending on the respective audience addressed. 
Each type of report has advantages and disadvantages: there are ostensibly 
no time constraints on written reports and data can be referred to as 
needed; however, there is a heavy dependence on writing and reading skills. 
Credibility and verbal skills of the presenter are important for oral 
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reports, which do provide for immediate feedback and clarification of 
audience questions. Oral reports should include specific sections in which 
the particular audience is interested. Generally oral reports are 
constrained by time and the amount of information presented may be limited. 

Tables and figures should sinplify and clarify information. When 
information is extensive, data tables and graphs or figures provide an 
effective sumrary. Both should be labelled clearly, described in texc or 
explained orally. Figures or graphs are generally used to quickly draw the 
audiences attention to a specific finding while tables provide detailed 
back-up data for further scrutiny and secondary analyses. 

5. Timing the Delivery o f Your Results 

If information is to be acted upon, it must be tiiiely. Audiences 
have their own schedule requirements; some may need reports at intervals 
throughout the program, while others may require only the final report. 
Periodic cenrounication with audience representatives and experience witi; 
program schedules, fiscal years, and decision cycles tell you when indivi- 
duals and groups need information. If a particular audience vill want an 
oral report, but will not be convening until several weeks after the 
evaluation report is written, it is advisable to provide an initial written 
sumnary to emphasize the importance of the information. 

B. Writing t he Report 

Report planr I ng is usually a group endeavor with the goal of collecting 
all relevant information and opinions concerning the program. Actual 
writing, however, is frequently assigned to one person, with assistance 
from consultants/reviewers as needed. This approach tends to produce a 
more coherent, stylistically smooth report. When there are multiple 
writers, it is advisable to secure the services of an editor or have one 
person on the writing team do a final edit. This is the case whether there 
is one or more multiple reports to be • .duced. 

Reports may be comprehensive or lifted to a specific area. The 
typical kinds of reports are: 

• Internal reports, sometimes written only for the files and 
used to further the evaluation process (e.g., questionnaire 
data and/or interpretation); 

• Progress reports with information on significant milestones; 
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• Interim reports focusing on events within an interval of 
time; and 

• Concluding reports that occur at the end of a program or a 
major interval (e.g. , end of first year) . Typos of 
concluding reports are: 

• A main report containing comprehensive informatl in about 
purposes, procedures, findings, and conclusions; 

• ^ Executive Sunmarv highlighting conclusions and 
sumtarizing more detailed information; these are often 
presented on a colored paper for quick access, and 

• Technical report containing detailed information about 
data collection strategies and analysis procedures, as 
well as, program information found in the main report. 

In addition, 

• Follow-up reports may be required for conferences, panels, or 
other types of meetings. 

• Announcements/press releases are a type of report that 
usually highlight key decisions. 

Figure 3 provides a basic evaluation report outline that should be 
tailored to meet individual audience needs and interests. 

The use of graphics within * report can enrich caimunication but if not 
used correctly can lead to false conclusions. Graphics should be simple, 
with legible print, whether projected or on paper copy, mteres. and mean- 
ing can be added by using symbols, e.g., dollar signs, profiles of faces; 
shading separation and color can highlight and facilitate comparisons. 
Perspective can enliven simple bar graphs but is difficult to dra-. 
correctly. 

C Using the Evaluation Results 

The production of a final evaluation report does not usually end the 
responsibilities of program personnel. Disseminating the program results 
is a critical link in the life cycle of a program, and very often it also 
serves as the initial link in a chain of program continuation/refunding. 
The audiences for many educational program evaluation reports are narrowly 
focused at the local school district level, with a school board, local 
superintendent, and administrators serving ac the primary audiences. For 
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Figure 3 

Sample Evaluation Report Outline 



I . Introduction 

A. Program description 
1* Background 

2 . Objectives 

B. Purpose of the evaluation 

1. Decisions to be made (e.g., program expansion, program 
improvement, etc.) and audience(s) . 

2. Evaluation questions 

3. Limitations and caveats 

II . Procedures 

A. Evaluation design(s) chosen (nay be different for different questions) 
and limitations of the design(s) 

B. Nature of sample and sampling procedures 

C. Information collection techniques 

1. Instrumentation (observation schedules, test, survey, etc.) 

2. Reliability and validity of instruments 

3. Field activities for collecting information 

D. Data Analysis 

1. Choice of technique (e.g., Analysis of Variance, chi- square, 
etc.) 

2. Rationale for choice of technique 

III. Results and Discussion of Results 

A. Interpretation of data analysis results relevant to each evaluation 
question 

B. Constraints under which the study was done 

IV. Conclusions and Recommendations 

V. Appendices 

Include copies of instruments used and any other documents 'irportant to 
understanding the evaluation. 

Executive Summary 

For most reports, an executive suitmary containing nore detail than the 
sunmary in the report but still a very condensed version of the report is 
desirable. 
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large scale national programs (e.g. , Chapter 1) , aggregated data are 
compiled for congressional review. The secondary/transition projects are 
very unlike the aforementioned programs. In many instances they serve 
historically neglected persons in our society. Their newness makes them 
innovative and their outcomes ultimately open doors to employment and 
higher education for handicapped persons. Benefits accrue to the partici- 
pants and spill over into the larger ccmnunity. Through secondary/ transi- 
tion programs the handicapped become contributing citizens with self- 
respect, the public gains a better understanding of the handicapped 
persons' capabilities, adaptive equipment becomes more commonplace, and 
agencies learn to work together for common goals. Evaluation results should 
be used to further the above outcomes, and the results should be examined, 
analyzed, and interpreted to improve the programs so the above outcomes 
automatically multiply as byproducts of successful programs. Distinguish- 
ing between internal ,md external uses of evaluation results is essential 
when planning for dissemination of results. 

Intern a l usas -valuation results are adninistratiffi and procedural . 
Information should oe provided for admiuatratora to use in decision- 
making, program planning, and problem solving. Which program component has 
the best cost/benefit ratio? When is the optimum time to approach 
employers in the local community? How can transportation be secured for 
students in rural areas? Wiat incentives can be used to maintain trainers, 
and how can parents become involved so that they promote the independence 
rather than dependence of their handicapped sons and daughters? How can 
staff participate in the development of the management plan so they feel 
some ownership and are responsive to deliverable dates and milestones? 

Procedural use of evaluation results refers to examining results with 
the goal of identifying program processes that can be improved, as well as 
those that are effective and need to be maintained or refined. Do longer 
training periods result in more longevity on the job? Does one type of 
adaptive equipment effect better skill transfer than another type? Does 
group counseling result in more positive social behavior than individual 
counseling? Do monthly interagency meetings promote program continuity 
better than newsletters and/or bi-monthly meetings? These types of ques- 
tions are best answxed by the teachers, trainers, and participants, i.e., 
those who are most familiar with the day-^o-day conduct of the program. 



External uses of evaluation results focus on providing information, 
changing attitudes and perceptions, encouraging acceptance and replication 
of the program, and meeting requirements mandated by funding sources. 
Clear, concise, and technically and gramrvatically accurate presentation of 
results enables stakeholders to have confidence in the information and to 
more clearly understand and interpret performance to the camuanity beyond 
the stakeholders. Recommendations for external audiences should be tied to 
data that support the recarmendations , and linkages should be established 
among equation, questions, data, information, and plausible action 
responses, stakeholder biases need to be acknowledged in anticipating 
positive or negative reactions to proposed progranraatic changes, so argu- 
ments that may be counter to the biases can be persuasively presented. The 
potential for publishing the results for an effective, well-conducted 
project in professional journals, newsletters and newspapers should be 
explored as part of the dissemination effort. Requests from readers for 
additional information can result in replication/adoption of the program in 
other locations. 

Because state and local funding sources are often constrained by a 
multiplicity of needs, it is critical that federal support for secondary/ 
transition projects continue until other funding sources are located. Thus 
the program reports for accountability are as important as others we have 
discussed. The Project officer, even though perhaps geographically distant 
from your project, should be apprised of your project's progress. The 
Project officer can be one of the program's strongest advocates, if kept 
informed. Copies of interim reports, press releases, and other documents 
are appropriate ccranunication devices. 

A statement from one of the Transition Institute's researchers is an 
appropriate end to this Handbook: 

"Evaluation responsibility is conmunication responsibility." 1 



1/ Stake, R.E. (1973), Evaluation Design, Instrumentation, Data Collection, 
and Analysis of Data in Educational Evaluation: Theory and Practice . 
In R. Worthen and R. Sanders, (Eds.), Educational Evaluation: Theory and 
Practice. Belmont, CA: Wadsworth Publishing, p. 315. 
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Appendix A 
Annotated Bibliography 



annotated Bibliography 



Borich, G.D. and Jeanelka, R.P. (1982). Programs and Systems: An Evaluation 
Perspective* New York: Academic Press. 

This book is about decisions , values , and systems. It is about how these 
concepts care together to create a network of ideas , activities , and 
responsibilities for program evaluation. Most of all, it is about the 
role that values play in binding together this network an'i in providing a 
s t a nd a r d for both planning and evaluating programs. This book is written 
for a broad array of readers interested in program planning, evaluation, 
or systems. 

Brinkerhoff, R.O., Brethower, D.M., Hluchyj, T., and Ncwakowaski, J.R. (1983). 

Program Evaluation- A Practitioner's Gu ide for Trainers and Educators 

(Sourcebook). Boston: Kluwer-Nijhoff Publishing. 

The Sourcebook is part of a package developed by the Evaluation Training 
Consortium (ETC) project at the Evaluation Center, Western Michigan 
University. The ETC project was funded by the U.S. Office of Special 
Education from 1972 to 1982 to develop program evaluation procedures for 
use by teacher educators. The Sourcebook contains chapters on guidelines, 
resources, and references for each of seven key evaluation functions. The 
presentation requires seme knowledge of evaluation methodology and 
terminology and lacks an index to topical areas. The other two documents 
in the series include a collection of 12 evaluation stories (Casebook) and 
a programed set of directions, worksheets, exanples, and checklists to 
help design an evaluation (Design Manual). 

Buros, O.K. (Ed.) (1938, 1940, 1949, 1953, 1965, 1972, 1980). Mental 
Measurments Yearbnrfrg. Highland Park, NJ: Gryphon Press. 

A series of eight yearbooks containing critical reviews of tests which are 
"designed to assist test users in education, industry, psychiatry, arid 
psychology to locate, choose, and use tests with greater ease and 
discrimination. M Each Yearbook contains up-to-date, comprehensive biblio- 
graphies of tests published recently in English-speaking countries, plus 
critical test reviews by measurement and other experts. Comprehensive 
test entries include listing of relevant test references in books, 
journals, and doctoral dissertations. Tests are grouped by function with 
references by author(s), title, publisher, and subject natter area. 

Buros, O.K (Ed.) (1974). Tests in Print tt. Highland Park, NJ: Gryphon 
Press. 



This book presents a comprehensive bibliography of all known tests 
published for use with English-speaking subjects and currently in print- 
ing. T^xe are 2467 test entries classified according to functional use 
(e.g., achievement batteries, personality tests, vocational tests). Tests 
also are referenced by publisher, title, and author(s), with a scanning 
index by subject natter area. 
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Fink, A. and Kosecoff, J. (1980). HOW To: Evaluate FdnraHnn Programs (A 
Ccnpi l ation of Ideas and Mafchnria Th*+- vfttf t) Vfeshington, DC: Capital 
Publications. 

This book contains many articles previously published in Capital Publica- 
tions monthly newsletter, "How To Evaluate Education Programs." It also 
includes chapters on "How To Set Evaluation standards," How to Choose An 
Evaluation Design, " and "How To Prove a Program Works." Some of the mor-e 
recent issues of the newsletter also include articles relevant to the 
evaluation of transition intervention projects as follows: 

o "How to Conduct a Survey" (May, 1981, Chapter 44) 

This issue discusses what a survey is, the different types of 
surveys, survey sampling considerations, hints for designing and 
administering a survey, survey pitfalls, and survey costs. 

o "How to Plan An Evaluation (November, 1981, Chapter 51) 

This issue presents a case study example of planning an evalua- 
tion, including the development of evaluation questions and 
standards, the evaluation design and sample, and data collection 
(including a data collection plan). 

o "How to Tell If A Program Meets Its Goals" (May, 1982, Chapter 
57) 

This issue defines goals, objectives and their various types 
(cognitive, affective, psychomotor) , and setting standards. 

o "How To Figure Costs For Evaluation Studies" (September. 1982 
Chapter 61) 

This issue shows how to prepare an evaluation budget, includina 
detail information. 

o "How to Evaluate In Action" (April, 1984) 

This issue presents sane evaluation "do's" and a "don't" based 
on a previous evaluation experience. The "do's" include 
involving program staff, flexibility, expecting the unexpected, 
and multiple reporting audiences. The "don't" is to not expect 
"everyone to care about the evaluation. " 

Joint Ccranittee on Standards for Educational Evaluation (1984). Standards for 
Eva l uat i ons of Edncflf Lanal Programs. BcajflStS M aterials. New York: 
McGraw-Hill Book Company. 

This book represents the results by representatives of twelve professional 
research and education organizations to develop a set of rigorous and 
useful standards for the evaluation of educational programs, projects, and 
materials. The resulting standards are referenced in two different ways: 
according to four evaluation standards (utility, feasibility, propriety, 
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and accuracy) or ten functional groupings (administering evaluation, 
analyzing information, butigeting evaluation, deciding whether to evaluate, 
defining the evaluation problem, designing evaluation, collecting informa- 
tion, contracting evaluation, reporting evaluation, and staffing 
evaluation) . 

Morris, L. L. (1978). Program Evaluation Kit . Beverly Hills: Sage. 

The Program^ Evaluation Kit is a set of books intended to assist people who 
are conducting evaluations of educational programs. The scope of its 
potential use is broad. Because it conprises a set of step-by-step proce- 
dural guides, the kit can advise a r erson conducting elaborate evaluations 
or it can help people as they gather, analyze, and interpret information. 
The kit also introduces and explains concepts and vocabulary canton to 
evaluation. # It is designed to be useful to people with extensive 
experience in evaluation as well as the novice. The kit consists of the 
following eight books. 

1. The Evaluator's Handbook 

2. How to Deal with Goals and Objectives 

3. How to Design a Program Evaluation 

4. How to Measure Program Implementation 

5. How to Measure Attitudes 

6. How to Measure Achievement 

7. How to Calculate Statistics 

8. How to Present an Evaluation Report 

Nowakowaki, J., Bunda, M.A., Wbrking, R., Bernacki, G., and Harrington, p. 
(1985). A Handbook of MnraHnnal Var iables: A Guide to Evaluation , Boston: 
Kluwer-Nijhoff Publishing. 

This handbook provides a comprehensive and well-organized set of questions 
and variables that might be appropriately addressed in evaluations of a 
wide range of educational programs and services. It is well grounded in 
both educational practice and eaucational theory. The level of language 
is oriented toward those who might be involved in planning an educational 
evaluation. 

Pietro, D.S. (Ed.) (1984). Evaluation Sourcebook for Privat e and Voluntary 
Organizations* New York: American Council of Voluntary Agencies for foreign 
Service. 

This Sourcebook is a practical guide for selecting appropriate tools for 
field-based program evaluation. It was written from a collaborative 
effort of private and voluntary organization practitioners experienced in 
evaluation, and is designed for policy-makers who need to coordinate and 
utilize evaluations. Section One serves as an introduction to evaluation, 
Section Two is a M how-to M section (covering topics such as goal-based 
evaluation and goal-free evaluation) , and Section Three discusses what to 
do when you need help. 



PitZ, G.F. and McKillip, J. (1984). Derision A nalysis for Program Era lnar.org . 
Beverly Hills: Sage. 
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This book describes in detail techniques that can be used for aiding deci- 
sions in the context of a program evaluation. The book is written for 
people who are unfamiliar with decision analysis. The authors assume only 
an ability to work with simple equations, equivalent to the skills 
required for a beginning graduate-level course in statistics (no knowledge 
of statistics is necessary) . Technical details which are unnecessary for 
someone concerned only with the application of decision analysis have been 
omitted. A short glossary is provided for a quick reference to the most 
important technical terms. 

Rutman, L. and Mowbray, G. (1983). Understandi ng Program Evaluation . Beverly 
Hills Sage. 

This is a guidebook to better management through the use of program 
evaluation. It is directed to two primary audiences: (1) managers in 
charge of national, state, and local social programs, and (2) students who 
one day may become program managers . It provides cases and questions in 
the appendix to provide guidance for planning and simulating evaluations. 
The chapters present information on the purposes of evaluation, steps in 
the evaluation process, planning the evaluation, measurement, research 
design, and using evaluations for decision-making. 

Sichel, J. L. (1982). Pro gram Evaluation Guidelines! A Research Handbook for 
ftjency Peraonnftl : New York: Human Sciences Press. 

This handbook is written for the nonresearcher who is faced with conduct- 
ing an evaluation of his/her program, it is a small, informal book which 
addresses practical issues - how to frame guiding questions, how to define 
the appropriate scope for an evaluation and "how to make sure you'll have 
something of use when you're done." 

Scriven, M. (1967). "The Methodology of Evaluation." In Tyler, R. w., Gauge, 
R. M. and Scriven, M. Perspective of Curri culum revaluation , aera Monograph 1. 
Chicago: Rand McNally and Company. 

This paper defines formative and summative evaluation and discusses the 
goals of evaluation versus the roles of evaluation. The paper also 
addresses amateur versus professional evaluation, evaluation studies 
versus process studies, evaluation versus estimation of goal achievement, 
intrinsic evaluation versus pay-off evaluation, some practical suggestions 
for cc^ination formative/suromative (hybrid) evaluations, criteria of 
educational achievement for evaluation studies, values and costs, and a 
number of other evaluation topics. 

Scriven, M. (1973), Goal-Free Evaluation. In E. R. House (Ed.), School 
Eva l uat i on; — The Politics and Pror-ess. Berkeley: McCutchan. 

Scriven discusses goal -free evaluation in the contents of summative and 
formative evaluation methods. His approach is unique, in that he utilizes 
a scenerio of an initial meeting between a client and a goal-free evalua- 
tor to illustrate his points. He provides additional explanation follow- 
ing the presumed dialogue. 
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Scriven, M. (1981). Evalution Thesaurus , Inverness , California: Edgepress. 



This book nay serve as a kind of minature text-cum-reference-guide to the 
field of evaluation. It is a smaller and less expensive guide than the 
encyclopedia, yet more comprehensive than glossaries since it is not 
restricted simply to educational evaluation or to program evaluation. It 
contains practical suggestions and procedures, contents and criticisms, as 
well as definitions and distinctions . 

Smith, N. L. (Ed.) (1981). New Techniques for Evaluation . Beverly Hills: 
Sage. 

This book provides an introduction to techniques that will enable the 
evaluator to deal with research, management, policy, value, and economic 
questions . The authors have tried to avoid being theoretical and abstract 
in favor of offering concrete alternatives to the practicing evaluator. 
The book contains five chapters, each dealing with a different technique 
or set of techniques for use in evaluation. Numerous exanples are 
presented. 

Smith, N. L. (Ed.) (1982). Omnunication <frraj-e qies in Evaluation , Beverly 
Hills: Sage. 

This book is about ccmrainication in evaluation and the ways that evalua- 
tion information can be coranunicated with greater clarity, inpact, and 
variety. The purpose of this volume is to change the way evaluator s think 
about the role of cxxmoinication in evaluation as well as to change the 
nature of the ccnitiunications they produce. Alternative techniques are 
presented, including: research briefs, graphic display procedures, stein 
and leaf displays, operational network displays, geographic displays, oral 
policy briefings, briefing panel presentations, adversary hearings, 
coranittee hearings, and television presentations. 

Struening, E. L. and Guttentag, M. (Ed.) (1975). Handbook of Evaluation 
Research - Volumes 1 & 7 Beverly Hills: Sage. 

One purpose of the Handbook is to provide evaluators with a sample of 
experts with whan they can ccmrainicate as they develop the crucial steps 
of their studies. More specifically, the Handbook provides the type of 
information that should lead to the consideration of alternative 
approaches to evaluation. The Handbook is also designed to serve as a 
textbook for courses in evaluation at the graduate level. The firs;- 
volume emphasizes strategies and methods of evaluation while the second 
volume reviews the literature of selected content areas. 

Stufflebeam, D. K. (1973). "An Introduction to the PDK Book: Educational 
Evaluation and Decision-Making." An address delivered at the Eleventh Annual 
PDK (Phi Delta Kappa) Synposium on Educational Research, Ohio state 
University, June 24, 1970. Reprinted in Wbrthen, B. R. and Sanders, J. R. 

Educational Evaluation; Theory and Practice . Wbrthington, Ohio: Charles A. 

Jones Publishing Company. 
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This paper presents Stufflebeara's CIPP Evaluation Model in its formative 
conception. This discussion includes a definition of evaluation and the 
terms implied in the definition, decision settings, the four types of 
decisions and the types of evaluation corresponding to them (Context, 
Input, Process and Product) . 

'St^S e ' G A 5" (1977K A"* Joint I nsemination Review p^i Trw^ ir. 
Washington, D.C.: U. S. Government Printing office. 

The Ideabook presents practical procedures for preparing presentations for 
tne Joint Dissemination Review Panel (JDRP) process. A new volume is 
expected in late 1986. 

Bacon™ 1 ' (1985) ' EvaluatincT InStDlCtiQDaJ EESgraBB Boston: Allyn & 

This book was written to provide the evaluator with the information and 
techniques to carry out instructional program evaluations in the schools, 
it has been written to be both practical and understandable. The author 
presents new and innovative approaches to evaluation, operational Guide- 
lines for doing formative, sunmative and ex post facto evaluations, 
strategies for qualitative evaluations, and suggestions for assessing 
teaching. Case studies are provided as well. 

Udinsky, B. F., Osterlind, S. J., and Lynch, S. W. (1981). Evaluation 
Resource Handbook; rrtt - hprina. Analysing. RepnrHng n^ a san Diego: edits. 

T^L^^tJ* 11 *^ 18 tow ards providing student or practi- 

tioner with the tools necessary to implement evaluation. The thirty-nine 
articles contained in the book have been written around topics which are 
ccroonly used ui gathering, analyzing, and reporting data. The articles 
are arranged into eight categories and are written in outline format. 
S * ?°nta^ real-world examples for application purposes, and 

each one is followed by a selected bibliography. The format is designed 
for persons concerned with implementing the evaluation. 

Wolf, R. M. (1984). Evaluation in Education New York: Praeger. 

This book attempts to furnish evaluation workers with a comprehensive, 
coherent, and integrated view of educational evaluation. It is intended 

"SoT^ 9 ^? 3 ?^ 1 ' , yet based on theor *- There is a fair amount of 
how to in the book, along with a theoretical base for the acplications . 
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Describing the Evaluation Setting* 



Chapter 2 



Project Clients 

60-72 mainstreamed learning disabled students who are enrolled in the General 
College of the University of Minnesota; all are taking programs that require 
successful completion of writing courses or demonstration of writing 
proficiency. 



Project Locale (Setting) 

General College °f the University of Minnesota, a 50-year-old open admissions 
college viitn a history of testing and disseminating curriculum for "by-passed 1 
populations, such as learning disabled students. 



Project Rationale/Goals & Objectives 

A multi-media curriculum in writing has shown promise in helping LD students 
acquire writing skills needed to produce clear, coherent, and concise written 
pieces. 

The project will develop test, implement, evaluate, and disseminate such a 
curriculum in order to increase the retention and academic performance levels 
of LD students. . . 



Project Description (Components, Activities, Resources, Problems) 

^e project will 1) create a research base of relevant literature, 2) trans- 
law research into a multi-media writing curriculum, 3) identify 20-24 LD 
students to participate in the project, 4) implement the curriculum in 2 
courses, 5) enroll participants in a career planning course, 6) evaluate 
project activities, 7) disseminate the evaluation results, and G) plan for the 
following year based on research and evaluation results. 



Project Timelines (To accomplish long & short term goals) 

This a 36-month project. Interim reports will be available at completion of 
months 4, 8, 16, 20, 28, and 32; annual reports will be available at end of 
months 12 and 24; a final report will be available in month 36. 



*NOTE: The description above is a very abbreviated example. A "real" 
description would be far more detailed. 
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Identifying Evaluation Audience* /Stakeholder* 



Chapter 2 



Key Aud 1 ences /Stakeho Iders Audience/Stakeholder Roles 



OSERS Staff 



Fundi ng agency 



LD C I ients 



Facu I ty /administrator* 
at other postseconda ry 

1 net i tut 1 one 



Consumers of services 



Potential replication sites; 
consumers of new knowledge 



Project Staff 



Imp I entente rs of curriculum 
and project as a whole 



Audi ence/Stakeho Ider 
Information Needs 



Progress of implementation 
Cost/expend itu res 
Genera I outcomes 



Individual progress and 

outcome 
Genera I outcomes 

Content and strategies used 

cu rr 1 cu I urn 
What worked/didn't work 
Exoend 1 tures for tasks 



Progress of Imp lementat i on 

problems and successes 
Ongoing inform tlon on what 

seems to be working/not 

working 
Effectiveness of both 

curriculum and dlssemina- 

tion strategy 



Audi ence/Stakeho Ider Hi ases 



Want data conducive to meta . 

* v m I u a 1 1 o n 
Want Congress to think money 

we I I spent 

Interested more In Individual 
progress/success 



Interested In adopting best 
parts of intervention at 
lowest possible cost 

Interest in theoretical basis 

for curriculum 

Want project as a whole to 
succeed 

Co not want that part of 
project he/she is respon- 
sible for to be evaluated 
as prob lem area 



E: The examples regarding information needs and stakeholder biases have been abbreviated in order to give several 
nples of stakeholders; however, the 4 groups of stakeholders cited here are but a part of a given project's potentia 



NOTE 
examp 

audiences/stakeholders. 
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Chapter 2 



Evaluation Rationale 



Evaluation Purpoee 



Evaluation Constraints 



Evaluation Products 



Rank 



To measure project 
ef f ect i veness/ outcomes 

To aid project implemen- 
tation and improvement 



Limited budget for external 
i nd I cator 

Limited tine to gather and 
review information/data on 
regular, frequent basis 



To meet funding agnecy Limited personnel knowledge- 

requirement for evaluation able about evaluation 



To provide information to Limited time to develop 
replication site separate document 



2 annual reports plus 
project final report 

Monthly progress report; 
2 annual reports; interval 
memoranda/meet i ngs re : 
problem areas 

Monthly progress reports 

sent to agency 
2 annual and 1 final report 

Separate document designed 
specifically to describe 
project and provide 
information outcomes , 
costs, etc. 



NOTE: These examples .re not .11 inclusive of evaluation purposes, constraints, or products, thes<< will of course vary 
from project to project. 
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Append!;; C 
^brksheets for Chapter 3 



Chapter 3 

Additional Types of Data Collection Procedures 



Axxetion Analysis 

Artifacts Analysis 
Case Studies 

Panels, Hearings 
Logs 

Simulations 
Sociograms 



Vfear or accumulation on physical 
objects 

By-products are observed 

Characteristics of selected persons 
in a project 

Opinions; Ideas 

Recorded behaviors and actions 

Person's behaviors in simulated 
settings 

Preferences for friends, work and 
social relationships 



Others; Systems Analysis, Advocate Teams, 
Tests, Q-Sorts, Time Series Analysis 



Judicial Review, Judgmental Ratings, 



Chapter 3 



Evaluation Management Plan 



Evaluation Activity 



Due Date or Time Span 



Staff Responsible for 
Each Act i v i ty 



Task 1: 

To develop survey 
instrument that iden- 
tifies gaps in service 
de I i very system 



End of Month 3 of project 



Research Assistant; 
Project Di rector 



Criteria for Completion 



Agreement of project staff 
thaw instrument will obtain 
all necessary data/ i ^f orma- 

t i on 



Task 2: 



Task 3: 



V AW 



Planning Guide for Data Collection and Analyala Procedurea 



Chapter 3 



Information Schedule (When, Sample (Kind and 

N##d » Ho "> Where) Size) Respondenta Inatrumentat ion Analyala 



Correlation between typi 
of training and client 
aatiafaction with 
p I acement 



Montha 10, 22, 

?nd 34; 
aurvey ; 
at job aite 



All project clienta Clienta 

(N=120) 



Sat 1 af action 
aurvey 



Corre I ati on 



Extent to which ctienta 
retain joba 



Month 36 of 

project; 
telephone contact 



All project clienta Clienta 



Telephone aurvey 



Percentagea of 

lat, 2nd, and 
3rd year c I ienta 
employed in 
original job at 
thia time 



Appendix D 
Worksheets for Chapter 4 
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Chapter 4 



Reporting and Using Evaluation Rasults 



Evaluation 
Quest 1 on 



Aud i tnct 



Was the project effec- 
tive In placing clients 
In competitive employment? 



Clients/ 
Fami lies 



Time I 1 nes/ 
Due Date 



DSERS Staff Month 36 



Month 36 



Data to Provide 



Format and Pre- 
sentation Methods 



Utl I Izatlon of 
Resu I ts 



Replication Month 36 
sites 



Percentage of 
c I 1 ents p I aced 1 n 
compet 1 t 1 ve 
emp I oyment 

Same as above 



Same as above 



Final Report 



Letter to al I 

c I lents/f ami I les 

Workshop (with 
overheads) for 
potent 1 a I rep I 1 - 
catlcn sites 



Part of Report 
to Congress 



Decision to repli- 
cate/not replicate 



